Combining Segmenter and Chunker for Chinese Word Segmentation

Our proposed method is to use a Hidden Markov Model-based word segmenter and a Support Vector Machine-based chunker for Chinese word segmentation. Firstly, input sentences are analyzed by the Hidden Markov Model-based word segmenter. The word seg-menter produces n-best word candidates together with some class information and confidence measures. Secondly, the extracted words are broken into character units and each character is annotated with the possible word class and the position in the word, which are then used as the features for the chunker. Finally, the Support Vector Machine-based chunker brings character units together into words so as to determine the word boundaries. 1 Methods We participate in the closed test for all four sets of data in Chinese Word Segmentation Bakeoff. Our method is based on the following two steps: 1. The input sentence is segmented into a word sequence by Hidden Markov Model-based word seg-menter. The segmenter assigns a word class with a confidence measure for each word at the hidden states. The model is trained by Baum-Welch algorithm. 2. Each character in the sentence is annotated with the word class tag and the position in the word. The n-best word candidates derived from the word seg-menter are also extracted as the features. A support vector machine-based chunker corrects the errors made by the segmenter using the extracted features. We will describe each of these steps in more details. Our word segmenter is based on Hidden Markov Model (HMM). We first decide the number of hidden states (classes) and assume that the each word can belong to all the classes with some probability. The problem is defined as a search for the sequence of word classes C = c 1 ,. .. , c n given a word sequence W = w 1 ,. .. , w n. The target is to find W and C for a given input S that maximizes the following probability: arg max W,C P (W |C)P (C) We assume that the word probability P (W |C) is constrained only by its word class, and that the class probability P (C) is constrained only by the class of the preceding word. These probabilities are estimated by the Baum-Welch algorithm using the training material (See (Manning and Schütze., 1999)). The learning process is based on the Baum-Welch algorithm and is the same as the well-known use of HMM for part-of-speech tagging problem, except that …